Misinterpreting a Failure to Disconfirm as a Confirmation: A Recurrent Misreading of Significance Tests

It is an old principle of significance testing that failure to disconfirm a hypothesis does not imply that this hypothesis is true. The inability to reject it could be due to the small size of the sample or to the high variance of the data. But, as shown below (see section IV), this principle is often violated, if not explicitly, then by implication. Thus the results may be stated correctly as "the data do not reject the hypothesis". But since the author then proceeds to write (particularly in the concluding section) as though his or her hypothesis had been confirmed the reader is left with the impression that the significance test has confirmed it. Essentially, the problem arises from confusing the correct procedure of using as the null the proposition that the maintained hypothesis is false with the incorrect procedure of using as the null the proposition that the maintained hypothesis is true. Presumably, one reason why the principle that failure to disconfirm does not imply confirmation of the hypothesis is because it seems to conflict with the widely accepted methodological rule that the only way a scientific hypothesis can ever be confirmed is by repeated failure to disconfirm it. Another reason may be the strong wish to obtain an unequivocal conclusion, even when the correct interpretation of the significance test only allows one to say the data do not reject the hypothesis outright, that they leave open the